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Department  of  Defense  (DoD)  Acquisition  Test  and  Evaluation  (T^E)  has  been  the 
gatekeeper  to  Major  Defense  Acquisition  Program  production  since  its  formalization  over  25 
years  ago.  Yet,  the  landscape  of  the  types,  methods,  and  sources  for  war  fighting  systems  has 
significantly  evolved.  The  Department  has  studied  and  recommended  action  for  Acquisition 
reform  for  decades,  only  tweaking  in  the  margins  for  TIAE.  The  time  is  right  for  DoD  to 
consider  a  new  approach  to  T^AE,  steering  away  from  the  “buy"  decision  to  the  more  relevant 
“acceptance"  and  “operational"  domains.  This  article  outlines  the  issues  and  proposes  a  new 
“Alpha-Omega”  map  for  T^AE  for  the  way  we  actually  procure  DoD  systems. 
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Department  of  Defense  (DoD)  leaders 
and  numerous  reform  studies  criti¬ 
cize  the  acquisition  process  for  its 
inability  to  control  spiraling  costs  and 
delays  in  getting  systems  to  the  user. 
In  response,  acquisition  reform  efforts  to  date  focus  on 
better  requirements,  efficient  resources  planning,  as 
well  as  increasing  feedback  and  accountability.^  DoD 
Test  and  Evaluation  (T6cE),  comprising  the  formal 
processes,  policies,  personnel,  equipment,  facilities,  and 
consumables  necessary  to  develop,  certify,  test  and 
evaluate  defense  systems  for  production,  has  faced 
much  of  this  criticism.  In  response,  T&E  reform 
focuses  on  process  streamlining,  reducing  overhead, 
and  further  integration  within  the  engineering  process 
to  better  support  the  “buy”  decision.  The  defense 
acquisition  process,  using  T&E  as  a  gatekeeper,  is 
relatively  unchanged  since  the  Packard  Commission 
recommendations.^  Yet,  the  methods  and  players  of 
DoD  acquisition  have  fundamentally  changed.  Just  as 
the  Pentagon  is  embracing  a  new  map  for  the 
application  of  military  power  based  on  an  active 
strategy  for  the  world,  as  it  is  today,  DoD  needs  a 
similar  active  strategy  for  who  is  best  served  by  T&E.^ 
This  article  examines  the  fundamental  influences  to 
Acquisition  T&E  and  the  results  of  major  studies  to 
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date  on  reforming  Acquisition  T&E,  and  concludes 
that  the  current  emphasis  on  “buying”  as  the  “raison 
d’etre”  for  T&E  must  be  replaced  with  a  new  two- 
tiered  framework  and  leadership  that  better  support 
both  contractual  necessity  and  operations  in  the  field. 
There  is  a  new  world  of  defense  systems  acquisition 
driving  the  need  for  a  new  map  for  T&E  based  on 
acceptance  and  operation,  which  I  suggest  are  the 
Alpha  and  Omega  of  a  new  T&E  order. 

Good  intentions 

Acquisition  T&E  follows  a  linear  engineering 
approach  to  reduce  risk,  build  insight  to  meet  contract 
delivery,  and  assess  the  delivered  configuration  in 
operational  environments,  verifying  that  the  system 
works  warranting  further  procurement.^  Acquisition 
T&E  is  Service-  and  system-centric,  managed  through 
dedicated  funding  and  contract  vehicles  with  both 
developmental  (DT)  and  operational  testing  (OT) 
supporting  the  “buy”  decision.  Pubhc  Law  delays  the 
decision  until  a  “Beyond  LRIP  Reporf’  is  pubUshed 
following  OT.^ 

Yet  for  all  its  formality,  T&E  plays  a  weak  role  in 
controlling  what  the  Department  actually  buys.  The 
current  DoD  Acquisition  policy  allows  initiation  of 
low-rate  initial  production  (LRIP)  just  after  Critical 
Design  Review.  What  test  results  are  available  support 
this  decision,  but  completion  of  testing  or  successful 
results  are  not  formal  prerequisites. 

There  is  no  DoD  “Approval for  Service  Use  (ASU)” 
decision  based  on  successful  test  results.  The  only  legal 
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hurdle  to  proceeding  beyond  LRIP  is  a  report  by  the 
Director  of  Operational  Test  and  Evaluation 
(DOT&E)  of  an  operational  test  where  the  test  plan 
was  approved  in  advance  by  DOTScE.^  Again,  as  with 
LRIP,  there  is  no  requirement  to  pass  the  test,  only  to 
conduct  it.  Acquisition  T&E  today  operates  within  a 
“buy”  construct  that  neither  demands  minimal  user- 
oriented  testing  before  beginning  the  production 
process  nor  formalizes  the  full  commitment  to 
production  through  successful  completion  of  testing. 

Emphasis  on  speed 

DoD  Acquisition  reform  since  the  mid-1990s 
emphasizes  faster  cycle  times  through  efficient  man¬ 
agement,  capitalizing  on  emergent  technologies,  field¬ 
ing  of  early  capabilities,  and  continuous  product 
evolutionary  cycles.  The  objective  is  to  maintain  a 
competitive  edge  by  getting  to  the  “buy”  decision 
faster.  Yet,  in  World  War  II  (WWII),  compelling  need 
put  emphasis  on  production,  with  T6cE  supporting  it. 
Production  changes,  additional  requirements,  and 
performance  shortfalls  based  on  experience  in  the  field 
were  the  foundations  for  block  upgrades.^  Over  60 
years  later,  the  Mine  Resistant  Ambush  Protected 
Vehicles  procurement  mimics  this  approach  with  an 
emphasis  on  TScE  supporting  production.® 

DoD  works  to  balance  procurement  for  both  a 
longer-term  near-peer  threat  as  well  as  near-continu¬ 
ous  engagement  against  a  less-defined  extremist 
threat.^  Senior  leaders  rely  on  the  current  acquisition 
construct  in  favor  of  one  that  responds  better  to  iU- 
defmed  threats,  requirements  that  grow  and  change 
rapidly,  and  technologies  that  evolve  many  times 
within  the  development  cycle. Today,  as  in  WWII, 
the  focus  is  getting  the  right  capability  to  the  field 
faster,  but  speed-to-user  is  not  enough  to  drive 
significant  change  in  T&E. 

A  new  environment,  really 

There  have  been  profound  changes  to  DoD  systems 
development  and  acquisition  brought  about  by  industry 
practice  and  by  government  policy  that  sets  the  new 
environment  for  T&E.^^  The  following  provides  an 
illustrative  snapshot: 

•  Requirements  process.  Requirements  have 
steered  away  from  the  primacy  of  technocrats, 
planners,  and  buyers  to  the  current  end-user.^^ 
Component  Commanders  present  unique  chal¬ 
lenges,  as  they  focus  on  near-term  needs  and  have 
different  visions  for  how  requirements  are  met 
and  how  advanced  technology  can  be  used.  These 
users  are  less  concerned  with  technology  nuances, 
industrial  influences,  and  specific  capabilities. 


Yet,  to  manage  successful  acquisition,  specificity 
is  critical  for  configuration  design  and  engineer¬ 
ing. 

•  Systems  development.  Systems  are  more  com¬ 
plex,  and  the  ability  to  characterize  fuUy  end-state 
performance  before  fielding  is  a  challenge.  Lead 
systems  integrators  have  increasingly  less  insight 
into  the  subsystems  they  are  integrating  and  thus 
less  confidence  in  understanding,  and  certifying 
to,  actual  systems-of-systems  performance.  The 
burden  increasingly  falls  back  to  the  Department 
to  resolve,  with  risks  not  only  from  increasingly 
complex  systems-of-systems  integration  with 
differing  maturity  but  also  from  the  globalization 
of  defense  industrial  capability. 

•  Industrial  base.  The  consolidation  of  the  defense 
industrial  base  through  the  1990s  has  left  DoD 
with  fewer  options  for  competitive  development 
of  major  capital  systems.  This  drives  systems  to 
take  on  inherent  design,  engineering,  production, 
and  management  practices  with  less  government 
insight.  Key  components  and  materials  will 
increasingly  be  available  only  from  foreign 
sources  with  subsequently  less  control  of  the 
design  and  engineering.^®  The  emphasis  shifts 
from  preproduction  to  as-delivered  product 
adequacy. 

•  Mismatched  acquisition  strategies.  The  recently 
signed  DoD  5000.02  Instruction  relies  on 
technologies  being  wrung  out  before  initiating 
development,  competitive  prototyping  used  to 
find  the  “best  of  breed,”  and  due  diligence 
through  T&E  before  production.  Yet,  technology 
evolves  too  quickly  to  tie  acquisitions  to  fixed 
baselines  with  initial  units  differing  in  perfor¬ 
mance  and  utility  from  those  later  on  in 
production.  Lead  systems  integrators  today 
deliver  systems  comprising  subsystems  in  various 
levels  of  maturity  and  product  life  cycles. 

•  Non-Service-unique  systems.  In  spite  of  the 
rhetoric,  DoD  does  not  buy  capability;  it  buys 
“things”  (systems)  that  are  married  to  others,  and 
users,  to  form  war-fighting  capabilities.  While 
DoD  is  procuring  more  “joint”  systems  than  ever 
before,  the  vast  majority  are  stiU  Service-centric 
asked  to  operate  in  ever  increasingly  joint 
environments.®^  It  has  become  more  difficult  to 
characterize  one  system’s  adequacy  for  its  own 
acquisition  decision  without  interdependent  sys¬ 
tems  that  in  and  of  themselves  are  of  varying 
maturity  levels. 

•  Networked  operations.  National  defense  strate¬ 
gy  reflects  ever-increasing  multi-Service  and 
Coalition  operations.®®  These  self-forming  oper- 
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ations  preclude  fully  understanding  interfacing 
systems  performance  or  concepts  of  operations  to 
support  an  adequate  operational  test  in  advance  of 
fielding  decisions.  While  the  Department’s  Test¬ 
ing  in  a  Joint  Environment  Roadmap  of  2004  set  a 
vector  to  lash  together  the  disparate  testing 
capabilities  within  the  Department  and  Industry, 
it  can  only  go  so  far  given  ever-changing 
configuration  baselines  and  unpredictable  alter¬ 
ations  of  netted  combat  systems. Testing  and 
subsequent  evaluations  will  focus  more  on  in¬ 
theater  assessments. 

•  Expanded  acquisition  authority.  Once  the 
domain  of  major  Service  Commands,  acquisition 
authority  has  spread  to  user  and  mission-centric 
organizations  such  as  Missile  Defense  Agency  and 
Special  Operations  Command,  each  with  their 
own  processes.  As  such,  systems  developers  will 
have  less  confidence  in  their  system’s  performance 
as  they  have  less  insight  into,  or  control  of, 
interfacing  systems.  T&E  will  less  likely  depend 
on  a  priori  knowledge  of  full  system  capability  and 
default  to  rudimentary  baseline  assessments. 

•  T&E  beyond  the  Program  Manager  (PM). 
System  complexity  and  interconnectivity  means 
that  testers  will  find  it  difficult  to  build  a  test 
scenario  that  characterizes  all  desired  perfor¬ 
mance  points  within  shorter  development  time 
frames.  Added  is  less  confidence  in  the  threat  or 
expected  concepts  of  operations  as  each  user  will 
likely  tailor  operations  to  their  own  needs. 
Acquisition  will  increasingly  rely  on  tests  outside 
the  PM’s  control  to  build  just  enough  insight  for 
the  decision  needed.  Capitalizing  on  other  data 
from  which  to  build  consensus  is  key  to 
Integrated  T&E  methodologies  of  the  revised 
DoD  5000.02.^^ 

•  Services  are  oversight.  Service  T&E  has  under¬ 
gone  massive  consolidation  since  the  mid-1990s, 
which  is  not  likely  to  be  reversed  anytime  soon. 
Since  2000,  the  U.S.  Army  has  consolidated  much 
of  its  T&E  organization  and  reduced  its  workforce 
between  half  and  two-thirds.  The  U.S.  Navy 
reduced  personnel  and  substantially  integrated  its 
prime  contractor/government  testing.  The  U.S.  Air 
Force  further  shifted  DT  control  to  prime  contrac¬ 
tors  with  commensurate  reductions  in  its  work¬ 
force.^^  The  burden  of  conducting  traditional 
Service  DT  has  fallen  more  onto  the  contractors 
as  part  of  the  product  acceptance  process. 

The  above  “snapshot”  shows  not  only  what  ways 
have  changed  when  the  Department  acquires  its 
systems,  but  also  that  most  aspects  cannot  be  addressed 


without  fundamental  change  in  T&E.  Much  of  what 
exists  in  today’s  “new”  methodologies  to  help  nudge 
the  acquisition  process  along  in  this  new  environment 
ignores  the  Acquisition  T&E  world  as  it  has  become. 
The  current  processes  quickly  succumb  under  the 
weight  of  the  endless  reviews  and  forums. 

No  real  change 

DoD  conducted  three  comprehensive  studies  on 
weapon  system  acquisition,  with  emphasis  on  T&E,  to 
include  the  Defense  Acquisition  Performance  Assess¬ 
ment  (also  known  as  [aka]  DAPA  Report),  the 
Defense  Science  Board  Task  Force  on  Developmental 
Test  and  Evaluation  (aka  DSB  Report  on  T&E),  and 
the  Joint  Defense  Capabilities  Study  (aka  Aldridge 
Study).  These  authoritative  studies  produced  a  myriad 
of  recommendations  for  T&E  and  Acquisition.  While 
each  report  had  its  emphasis,  their  findings  and 
recommendations  for  T&E  were  generally  similar 
and  grouped  into  four  broad  thrusts: 

•  gain  organizational  efficiencies  by  blurring  the 
distinction  between  DT  and  OT, 

•  push  discovery  earlier  in  the  process  through 
more  rigorous  testing  up  front, 

•  increase  transparency  and  streamline  process 
overhead,  and 

•  better  utilize  the  planning  and  acquisition 
processes  for  joint  war  fighting  needs. 

All  three  studies  hoped  to  efficiently  push  T&E  to 
better  support  the  “buy”  decisions  through  process 
streamlining  and  combining  DT  and  OT  events  where 
possible  as  part  of  an  integrated  T&E  framework 
feeding  a  continuous  thread  of  discovery.  These  reports 
form  the  basis  of  the  DoD  5000  Integrated  T&E 
strategy.  Yet,  “integration”  is  fundamentally  an  effi¬ 
ciency  exercise  of  questionable  purpose  as  early-on 
schedule  and  cost  avoidance  are  lost  through  later 
rework  and  retest. 

Unfortunately,  neither  report  reassessed  T&E’s  role 
or  its  customer.  While  testing  is  fundamental  to 
systems  engineering  and  contractual  compliance. 
Acquisition  T&E  is  seen  as  a  “speed  bump”  to 
procurement.  In  trying  to  serve  many  masters,  T&E 
became  costly,  less  efficient,  and  its  reports  of 
questionable  utility  to  both  buyers  and  users. 

A  future  of  many  masters 

Future  Acquisition  T&E  must  support  two  acqui¬ 
sition  extremes,  the  quick-reaction,  less-defined  threat 
and  the  long-term,  near-peer  threat.^®  It  must  also 
support  near-term  contractual  necessities  as  well  as 
longer-term  product  life-cycle  processes.  The  emphasis 
is  on  early  capability  delivery  for  initial  fielding.  For 


30(4)  •  December  2009  475 


Axiotis 


many  complex  systems,  the  Department  will  only  begin 
to  understand  what  it  has  received  once  it  is  delivered 
and  operating  in  the  field.  The  Missile  Defense 
Agency  recognized  this  and  developed  T&E  processes 
to  support  dedicated  knowledge  points  that  now  form 
the  basis  for  the  revised  DoD  5000.^° 

As  in  WWII,  we  find  ourselves  with  users  forward 
deployed  and  persistently  engaged  and  needing  75 
percent  solutions  in  months.  DoD  must  be  more 
efficient  and  effective  in  getting  information  to  the 
user  and  feedback  from  operations  in  the  field. 

A  new  model 

Utility,  not  buying 

Acquisition  T&E  must  focus  on  its  mission,  not 
function,  to  support  Acquisition  and  system  complexity 
as  they  are  today.  T&E  must  uncover  critical  risks  prior 
to  initiating  a  program  or,  once  begun,  build 
knowledge  to  trade  off  risk.  The  focus  today  is  to 
provide  capability  as  soon  as  it  is  ready,  with  T&E  the 
primary  mechanism  for  fielding  the  right  capability  at 
the  right  time.  This  new  model  proposes  it  be  separate 
from,  but  affiliated  to,  the  buying  decision. 

While  we  acquire  systems  through  the  buying 
process,  it  is  capabilities  based  on  aggregates  of 
constantly  evolving  systems  that  are  delivered  to  the 
user.  Authority  to  initiate  development  has  become  the 
initial  production  approval  point  reflecting  the  national 
commitment  it  is.  The  Acquisition  process  is  no  longer 
the  tidy  affair  it  once  was.  Yet,  it  is  how  DoD  responds 
that  is  the  basis  for  a  new  T&E  model,  which  shifts 
emphasis  from  “buying”  to  the  more  relevant  product 
acceptance  and  operational  domains. 

An  “Alpha-Omega”  model 

The  new  model  for  T&E  shifts  the  emphasis  from 
buying  to  two  basic,  but  not  necessarily  sequential, 
domains.  The  first  includes  activities  to  characterize 
sufficiently  systems  in  support  of  contractual  necessi¬ 
ties,  management,  and  initial  fielding  decisions — the 
world  of  acceptance  tests  or  “Alpha  Tests.”  The  second 
includes  the  operational  assessments  made  some  time 
later  to  assess  mission  value  added  over  the  fielding  life 
cycle  or  “Omega  Tests.” 

The  vision  is  a  T&E  process  that  accelerates  the 
delivery  of  initial  DoD  capability  by  developers,  while 
ensuring  continuous  evaluation  of  performance  in  the 
field  for  current  operations  and  future  capability 
development.  This  approach  supports  acquisition  and 
life  cycle  activities  such  as  the  Department’s  Perfor¬ 
mance-Based  Logistics  and  Training. 

Alpha  testing 

Alpha  Testing  events  are  necessary  to  meet  contrac¬ 
tual  requirements  by  capturing  initial  baseline  capabil¬ 


ity  for  Service  use.  “Alphas”  comprise  aU  initial 
experiments,  contractor  development  tests,  quality 
tests.  Service-unique  interface  and  environmental 
compliance  tests,  security  and  accreditation  tests.  They 
are  the  necessary  blending  of  contractor  tests  (CT) 
used  to  support  delivery  to  the  government  with  the 
traditional  Service-oriented  interface  testing  (DT) 
performed  later  on.  Alpha  Testing  is  a  continuous 
aggregate  of  events,  not  necessarily  fiilly  completed 
events  or  pass-fail  by  their  structure.  Alphas  are 
“owned”  by  a  much  broader  community  of  stakeholders 
and  not  necessarily  under  any  one  single  agent’s 
control.  Their  results  form  the  basis  for  decision  gates 
ultimately  for  service  use.  Alpha  Tests  provide  the  basis 
for  understanding  delivered  items  at  the  time  of 
delivery,  not  necessarily  against  a  priori  baseline 
parameters. 

An  Alpha  Test  construct  capitalizes  on  aU  existing 
data  sets,  whether  or  not  contractor  derived,  and  is  not 
throttled  by  concerns  over  the  color  of  money, 
contracts,  or  ownership.  It  feeds  on  other  Service 
efforts,  direct  and  indirect  PM  efforts,  training  and 
fielding  activities.  Alphas  fiU  the  bin  of  system 
knowledge  regardless  of  source.  Alphas  provide  the 
PM,  and  those  of  affiliated  efforts,  the  freedom  to 
select  the  appropriate  data  from  which  to  argue  the 
case  for  delivery,  up  to  and  including  Approval  for 
Service  Use  (ASU).  Where  there  is  lack  of  data,  the 
PM  is  obligated  to  fill  the  void  or  ensure  that  others  do 
their  share  to  help  build  the  case  for  ASU. 

An  Alpha  approach  requires  involvement  by  cus¬ 
tomers,  users,  test  and  oversight  agents  for  insight  and 
advice  where  practical  or  necessary  given  their  control 
over  ASU.  Less  oversight  is  required  during  Alpha 
Testing  as  the  burden  faUs  on  the  PM  to  build  the  case 
to  deliver  the  incremental  capability  to  the  next  user  or 
integrator  in  the  chain.  This  methodology  is  consistent 
with  that  used  by  sub-tier  vendors  delivering  subsys¬ 
tems  to  the  lead  systems  integrators  and  consistent 
with  the  Department’s  Systems  Engineering  Guide.^^ 

Omega  testing 

Omega  Tests  are  those  scripted  and  unscripted, 
supervised  and  unsupervised  demonstrations  of  systems 
operation  in  the  field.  Users,  operational  test  agents, 
oversight,  training,  logistics,  and  doctrine  agents  focus 
on  system  utility  and  are  less  concerned  with  the 
buying  decision.  Omega  Testing  capitalizes  on  data 
and  experience  in  the  field,  not  as  pass-fail  since  the 
Department  has  long  since  committed  to  the  program, 
but  to  build  on  the  baseline  understanding  of 
capabilities  and  limitations  at  ASU.  Omega  feedback 
also  forms  the  basis  for  the  next  capability  increment  or 
decision  to  move  on  to  new  capabilities.  Data  and 
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insight,  through  formal  reports,  assessments,  or 
observations,  are  provided  to  the  community  at  large 
to  include  operations  research,  requirements  genera¬ 
tors,  product  life-cycle  managers,  Program/Project 
Managers,  oversight  entities  such  as  Service  Chiefs 
and  DOTScE,  and  training  and  doctrine  agents. 

A  significant  issue  using  today’s  OT&E  construct  is 
pegging  deficiencies  uncovered  in  complex  systems-of- 
systems  tests  for  a  product-centered  acquisition 
process.  An  Omega  strategy  broadens  the  responsibil¬ 
ity  as  these  events  are  funded  through  a  myriad  of 
single  and  combined  sources  to  include  Component 
Commanders,  Training  and  Doctrine  Commands, 
Research,  Logistics  and  Engineering  Activities,  Intel¬ 
ligence  Agencies,  Programs,  and  other  Service  Acqui¬ 
sition  agents. 

This  approach  expands  the  community  of  Omega 
agents  far  beyond  that  limited  by  the  Service 
Operational  Test  Activity  (OTA)  and  removes  the 
“black  hat”  image  of  today’s  operational  testers.  There 
would  be  less  concern  by  Acquisition  principals  that 
OTA  input  blurs  the  role  between  system  buying  and 
fielding.  Nevertheless,  U.S.C.  Title  10  must  be 
revisited  given  the  requirement  to  conduct  an  OT, 
and  for  DOT&E,  an  independent  operational  assess¬ 
ment  must  be  made,  prior  to  proceeding  beyond  LRIP. 
It  is  likely  DoD  will  need  consensus  with  Congress  to 
either  formalize  a  supervised  period  of  Alpha  testing 
on  basic  systems  to  support  independent  reporting  or 
use  the  first  Omega  evaluation  as  the  gatekeeper  to 
further  cross-Service  capabilities.  The  latter  would 
seem  more  appropriate  as  Congress  and  the  DoD  get  a 
better  picture  of  capabilities  fielded  and  future  needs 
with  effectiveness  judged  through  a  broader  evaluation 
lens. 

Organizing  to  the  Alpha  and  Omega 

Service  field  activities  would  continue  to  function  as 
life-cycle  agents  and  as  centrally  or  directly  funded 
Alpha  testers,  supporting  any  Alpha  event  whether 
Service-specific  or  at  Contractor  sites.  Alpha,  being 
nonpartisan,  can  be  managed  either  before  formal 
program  initiation,  during  program  phases,  or  as  part 
of  postproduction  life-cycle  support.  Much  of  this 
structure  is  already  in  place  as  test  personnel  at 
Department  Major  Range  and  Test  Facility  Base 
activities  are  direct  customer-funded  operations. 

Service  OTAs,  freed  from  the  grip  of  the  acquisition 
process,  support  customers  of  all  types.  OTA  and 
Omega  would  be  funded  through  a  much  broader  array 
of  customers  less  tied  to  programs.  The  expeditionary 
OTA,  or  other  agents  tapped  for  such  roles,  deploy  to 
theaters  of  operations  or  specific  test  sites  to  act  as  user 
test  or  evaluation  agents.  A  much  smaller  senior  cadre 


would  be  reserved  for  overseeing  Alpha  events 
supporting  ASU  decisions  through  working  arrange¬ 
ments  with  program  offices.  Their  portfolio  of 
products  and  services  would  be  greater  than  current 
program-centric  assessments.  The  OTAs  would  be 
managed  by  the  Services,  overseen  by  DOT&E,  and 
free  to  expand  their  operations  worldwide  to  include 
foreign  systems.  This  new  and  expanded  role  sets  the 
OTA  on  a  path  to  supporting  future  war  fighting 
capability. 

The  emphasis  is  on  empowering,  with  responsibility 
based  on  a  closer  working  relationship  between  the 
developers  and  users.  The  Alpha-Omega  strategy  relies 
on  three  simple  rules  by  which  to  frame  progress  and 
argue  for  ASU  when  appropriate. 

1.  What  war  fighting  capability  is  provided  (i.e.,  not 
the  “thing”  being  procured)? 

2.  To  what  degree  does  it  work,  and  how  do  you 
know  (i.e.,  capabilities/limitations  as  delivered)? 

3.  What  are  the  impacts  to  other  systems  (i.e.,  risk 
assessment  across  the  Doctrine,  Organization, 
Training,  Materiel,  Leadership  and  education. 
Personnel,  and  Facilities  domains)? 

Empowered  testers 

Testers  and  evaluators  increase  their  impact  on  new 
program  vectors.  The  emphasis  is  not  on  whether 
systems  are  good  enough  to  buy  (as  they  are  already 
being  bought)  but  rather  what  new  vector  must  be  set 
based  on  performance  and  deficiencies  observed. 
OTAs  plan,  manage,  and  oversee  Omega  Tests  as  well 
as  assess  capability  in  the  field,  working  with  the  users 
to  vet  future  capabilities,  upgrades,  or  changes  to 
doctrine  and  concepts  of  operations.  A  new  Joint 
Omega  Executive  provides  both  independent  and 
collaborative  insight  of  systems-of-systems  operations 
in  the  field  to  support  capability  increments. 

The  right  oversight 

The  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics — through  either  the  Direc¬ 
tor  for  Systems  Engineering  or  the  new  Director  for 
Developmental  T&E^^ — oversees  Alpha  activities, 
ensuring  that  adequate  insight  and  progress  supports 
(along  with  user  input)  the  decision  when  capabilities 
are  fielded.  This  leader  would  also  ensure  sufficient 
capacity,  training,  and  capability  exists  at  T&E 
facilities.  The  DOT&E  would  oversee  aU  Omega 
testing,  advise  on  operational  realism  for  Alpha  events, 
and  continue  to  report  independently  to  Congress. 
Oversight  agents  would  focus  less  on  technical  detail 
and  more  on  validating  that  the  achieved  capability  is 
usable  and  understood  by  the  users. 
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A  new  Test  and  Evaluation  Master 
Plan  (TEMP) 

A  cottage  industry  has  been  built  around  TEMP 
production  to  document  the  PM  and  OTA  commit¬ 
ment  for  T8cE.  Yet,  in  the  pace  of  today’s  program¬ 
matic  change,  the  TEMP  becomes  rapidly  outdated. 
Under  this  process,  the  TEMP  merely  outlines  the 
PM’s  top-level  strategy  for  the  next  ASU  decision 
focusing  on  system  maturity,  external  resources,  and 
likely  data  collection  points  for  ASU. 

Conclusions 

The  recently  revised  DoD  instruction  on  acquisition 
strengthens  the  primacy  of  fielding,  through  acquisi¬ 
tion,  with  T&E  primarily  supporting  the  latter.  This 
article  proposes  a  new  map  for  Acquisition  T6cE 
supporting  today’s  persistent  engagement  as  well  as  the 
next  near-peer  threat.  The  DoD  cannot  wait  for 
optimal  solutions  before  fielding  capabilities  or  rely 
solely  on  T&E  as  its  gatekeeper.  This  new  Alpha- 
Omega  strategy,  based  on  acceptance  testing  for 
delivery  and  operational  use  evaluations  in  the  field, 
is  on  par  with  Acquisition  as  it  is  today,  not  on  how  we 
wish  it  to  be.  This  strategy  recognizes  and  accepts 
T&E’s  core  role  in  engineering  and  contract  compli¬ 
ance,  as  well  as  T&E’s  ultimate  customer — the  user. 

This  article  examines  how  the  acquisition  environ¬ 
ment  has  changed  and  how  the  process  itself  has 
evolved  as  it  continues  to  adapt  to  this  new  reality. 
Nevertheless,  recent  authoritative  studies  on  T&E 
have  not  recognized  these  fundamental  changes  in  the 
landscape  and  have  only  recommended  modest  changes 
to  T&E  processes  to  speed  it  up  a  bit  and  make  it  cost 
a  little  less.  T&E  must  emerge  from  its  relegated 
shadows  in  acquisition  to  support  a  new  customer  set. 
The  Alpha-Omega  strategy  hopes  to  change  this  by 
shifting  the  traditional  OTA  role  out  of  the  “buy” 
process  into  the  more  relevant  fielding  process  as  the 
agent  of  choice  for  a  much  wider  set  of  customers  to 
include  not  only  Service  acquisition  and  life-cycle 
agents,  but  also  Component  Commanders,  Trainers 
and  Doctrine  agents,  and  Requirements  developers. 

The  Alpha-Omega  strategy  for  T&E  supports 
bringing  capability  to  the  field  faster,  with  better 
understanding  of  capabilities  and  limitations,  across  a 
broader  set  of  systems  of  systems  than  current 
methodologies,  streamlined  or  not,  can  ever  do.  The 
time  is  right  for  fundamental  change.  □ 

George  Axiotis  currently  leads  the  Integrated  Resources 
Analysis  Team  for  the  director.  Operational  Test  and 
Evaluation  (DOT^^E)  providing  Congressionally  man¬ 
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resources  investments.  Mr.  Axiotis,  a  recent  graduate  of  the 
Industrial  College  of  the  Armed  Forces,  has  over  26  years 
in  defense  acquisition  and  T^E  having  been  program, 
project,  and  test  manager  on  four  Defense  Acquisition 
programs,  a  program  sponsor  as  well  as  T^^E  advisor  to 
two  program  executive  ojficers.  Before  coming  to  the 
Pentagon,  George  was  director  of  the  Naval  Sea  Systems 
Command  Test  and  Evaluation  Office.  E-mail:  george. 
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Test  and  Evaluation,  Office  of  the  Secretary  of  Defense 
Derrick  Hinton,  Principal  Deputy  Director,  Test  Resource 
Management  Center,  Office  of  the  Under  Secretary  of 
Defense,  Acquisition  Technology  and  Logistics 
Ernest  A.  Seglie,  Ph.D.,  Science  Advisor,  Director, 
Operational  Test  and  Evaluation,  Office  of  the  Secretary 
of  Defense 

TO  REGISTER  ON  LINE  VISIT 

WWW.ITEA.ORG 


Technical  Tracks 

■  LVC  in  the  Urban  Environment 

■  Unmanned  and  Autonomous  System  Testing  (UAST) 
in  the  LVC  Environment 

■  W&A  of  LVC  Distributed  Environments  for  Testing 

■  Future  Trends  &  Needs  for  Distributed  T&E  Infrastructure 

■  Other  M&S  Topics 

Tutorials 

■  Synthetic  Natural  Environments  for  M&S  Support  of  T&E 

■  Sample  Size,  Confidence,  and  Designed  Experiments: 
Related  Fundamental  Concepts  in  Test  and  Evaluation 

■  Good  Enough  W&A 

■  Efficient  Simulation  Using  DOE  Methods 

■  The  Basics  of  the  M&S  W&A  Process 

■  The  Test  and  Training  Enabling  Architecture  (TENA) 
Enabling  Technology  For  the  Joint  Mission 
Environment  Test  Capability  (JMETC)  in  Distributed 
Live,  Virtual,  and  Constructive  (LVC)  Environments 

Extubits 

Your  company  or  government  organization  will  want  to 
take  advantage  of  the  premium  space  that  is  available 
for  you  to  display  and  demonstrate  products  and  services 
for  the  test  and  evaluation  community.  To  obtain  an 
application  to  exhibit  or  to  see  the  floor  plan,  visit  the 
ITEA  website. 

Sponsorship 

Your  sponsorship  dollars  will  defray  the  cost  of  this 
event  and  support  the  ITEA  scholarship  fund,  which 
assists  deserving  students  in  their  pursuit  of  academic 


disciplines  related  to  the  test  and  evaluation  profession. 
For  more  information  on  the  benefits  of  sponsorship,  or  to 
obtain  a  pledge  form,  please  visit  the  ITEA  website. 

Golf  ToumameiTt 

Monday,  January  11  at  the  Butterfield  Trail  Golf  Club. 
Butterfield  Trail  was  recently  voted  "Top  10  New  Courses 
You  Can  Play"  by  GOLF  Magazine.  Contact  Mr.  Vernon 
Diaz,  NewTec,  White  Sands  Missile  Range,  575-678-2145. 
Look  for  the  flyer  to  be  published  at  www.itea.org. 

Lodging 

The  Wyndham  El  Paso  Airport 
2027  Airway  Blvd.,  El  Paso,  Texas  79925 
915-778-4241  /  800-742-7248 
www.wyndhamelpaso.com 

The  Wyndham  is  located  immediately  outside  (walking 
distance)  to  the  El  Paso  International  Airport.  A  block  of 
rooms  is  available  for  the  2010  government  per  diem  rate 
with  a  cut-off  date  of  December  30, 2009. 

Ctmference  Planning  Committee 
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Mr.  Hank  Newton  •  575-835-7270  «  hnewton@aoc.nrao.edu 
Exhibits  &  Sponsorships; 

Mr.  Bill  Dallas  *  703-631-6226  •  wdallas@itea.org 
Registration: 

Mrs.  Jean  Shivar  *  703-631-6225  *  jean@itea.org 
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